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ABSTRACT 



A system allowing a user of a browser program on a 
computer connected to an open distributed hypermedia 
system to access and execute an embedded program obj ect. 
The program object is embedded into a hypermedia doCu- 
ment much like data objects. The user may select the 
program object from the screen. Once selected the program 
object executes on the user's (chent) computer or may 
execute on a remote server or additional remote computers 
in a distributed processing arrangement. After launching the 
program object, the user is able to interact with the object as 
the invention provides for ongoing interprocess communi- 
cation between the application object (program) and the 
browser program. One application of the embedded program 
object allows a user to view large and complex multi- 
dimensional objects from within the browser's window. The 
user can manipulate a control panel to change the viewpoint 
used to view the image. The invention allows a program to 
execute on a remote server or other computers to calculate 
the viewing transformations and send frame data to the 
client computer thus providing the user of the client com- 
puter with interactive features and allowing the user to have 
access to greater computing power than may be available at 
the user's chent computer. 

10 Claims, 9 Drawing Sheets 

Microfiche Appendix Included 
(4 Microfiche, 375 Pages) 
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DISTRIBUTED HYPERMEDIA METHOD 

FOR AUTOMATICALLY INVOKING 
EXTERNAL APPLICATION PROVIDING 
INTERACTION AND DISPLAY OF 
EMBEDDED OBJECTS V^ITHIN A 
HYPERMEDIA DOCUMENT 

NOTICE REGARDING COPYRIGHTED 
MATERIAL 

A portion of the disclosure of this patent document 
contains material which is subject to copyright protection. 
The copyright owner has no objection to the facsimile 
reproduction by anyone of the patent document or the patent 
disclosure as it appears in the Patent and Trademark OfiSce 
file or records, but otherwise reserves all copyright rights 
whatsoever. 

BACKGROUND OF THE INVENTION 

This invention relates generally to manipulating data in a 
computer network, and specifically to retrieving, presenting 
and manipulating embedded program objects in distributed 
hypermedia systems. 

Computer networks are becoming increasingly popular as 
a medium for locating and accessing a wide range of data 
from locations all over the world. The most popular global 
network is the Internet with millions of computer systems 
connected to it. The Internet has become popular due to 
widely adopted standard protocols that allow a vast inter- 
connection of computers and localized computer networks 
to communicate with each other. Computer systems con- 
nected to a network such as the Internet may be of varying 
types, e.g., mainframes, workstations, personal computers, 
etc. The computers are manufactured by different companies 
using proprietary hardware and operating systems and thus 
have incompatibilities in their instruction sets, busses, 
software, file formats and other aspects of their architecture 
and operating systems. Localized computer networks con- 
nected to the Internet may be incompatible with other 
computer systems and localized networks in terras of the 
physical layer of communication including the specific 
hardware used to implement the network. Also, different 
networks use differing, incompatible protocols for transfer- 
ring information and are not able to communicate with each 
other without a translation mechanism such as a "gateway*'. 

The Internet provides a uniform and open standard for 
allowing various computers and networks to communicate 
with each other. For example, the Internet uses Transfer 
Control Protocol/Internet Protocol ("TCP/IP') that defines a 
uniform packet-switched communication standard which is 
ultimately used in every transfer of information that takes 
place over the Internet. 

Other Internet standards are the HyperText Transmission 
Protocol ("HTTP") that allows hypertext documents to be 
exchanged freely among any computers connected to the 
Internet and HyperText Markup Language ("HTML") that 
defines the way in which hypertext documents designate 
links to information. See, e.g., Berners-Lee, T. J., "The 
world-wide web," Computer Networks and ISDN Systems 
25 (1992). 

A hypertext document is a document that allows a user to 
view a text document displayed on a display device con- 
nected to the user's computer and to access, retrieve and 
view other data objects that are linked to hypertext words or 
phrases in the hypertext document. In a hypertext document, 
the user may "cUck on," or select, certain words or phrases 
in the text that specify a link to other documents, or data 
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objects. In this way, the user is able to navigate easily among 
data objects. The data objects may be local to the user's 
computer system or remotely located over a network. An 
early hypertext system is HyperCard, by Apple Computer, 
5 Inc. HyperCard is a standalone system where the data objects 
are local to the user's system. 

When a user selects a phrase in a hypertext document that 
has an associated link to another document, the linked 
document is retrieved and displayed on the user's display 
30 screen. This allows the user to obtain more information in an 
efficient and easy manner. This provides the user with a 
simple, intuitive and powerful way to "branch off*' from a 
main document to learn more about topics of interest. 

Objects may be text, images, sound files, video data, 
documents or other types of information that is presentable 
to a user of a computer system. When a document is 
primarily text and includes links to other data objects 
according to the hypertext format, the document is said to be 
a hypertext document. When graphics, sound, video or other 
media capable of being manipulated and presented in a 
computer system is used as the object linked to, the docu- 
ment is said to be a hypermedia document. A hypermedia 
document is similar to a hypertext dociunent, except that the 
user is able to click on images, sound icons, video icons, etc., 
that link to other objects of various media types, such as 
additional graphics, sound, video, text, or hypermedia or 
hypertext documents. 

FIG. 1 shows examples of hypertext and hypermedia 
documents and links associating data objects in the docu- 
ments to other data objects. Hypermedia document 10 
includes hypertext 20, an image icon at 22, a sound icon at 
24 and more hypertext 26. FIG. 1 shows hypermedia docu- 
ment 10 substantially as it would appear on a user's display 
screen. The user is able to select, or "click" on icons and text 

35 

on a display screen by using an input device, such as a 
mouse, in a manner well-known in the art. 

When the user clicks on the phrase "hypermedia," soft- 
ware running on the user's computer obtains the link asso- 
ciated with the phrase, symbolically shown by arrow 30, to 
access hypermedia document 14. Hypermedia document 14 
is retrieved and displayed on the user's display screen. Thus, 
the user is presented with more information on the phrase 
"hypermedia." The mechanism for specifying and locating a 
linked object such as hypermedia document 14 is an HTML 
"element" that includes an object address in the format of a 
Uniform Resource Locator (URL). 

Similarly, additional hypertext 26 can be selected by the 
user to access hypertext document 12 via link 32 as shown 

50 in FIG. 1. If the user selects additional hypertext 26, then the 
text for hypertext document 12 is displayed on the user 
screen. Note that hypertext document 12, itself, has hyper- 
text at 28, Thus, the user can click on the phrase "hyper- 
media" while viewing document 12 to access hypermedia 

55 document 14 in a manner similar to that discussed above. 
Dociunents, and other data objects, can be referenced by 
many links from many different source documents. FIG. 1 
shows document 14 serving as a target link for both docu- 
ments 10 and 12. A distributed hypertext or hypermedia 

60 document typically has many links within it that specify 
many different data objects located in computers at different 
geographical locations connected by a network. Hypermedia 
document 10 includes image icon 22 with a link to image 16. 
One method of viewing images is to include an icon, or other 

65 indicator, within the text. 

Typically, the indicator is a very small image and may be 
a scaled down version of the full image. The indicator may 
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be shown embedded within the text when the text is dis- display(LCD),etc.), local storage (hard disk drive, etc.), and 
played on the display screen. The user may select the other components. Typically, small computer 104 is con- 
indicator to obtain the full image. When the user clicks on nected to a larger computer, such as server A at 106. The 
image icon 22 browser software executing on the user's larger computer may have additional users and computer 
computer system retrieves the corresponding full image, 5 systems connected to it, such as computer 108 operated by 
e.g., a bit map, and displays it by using external software "ser 110. Any group of computers may form a localized 
called a "viewer." This results in the full image, represented network. A localized network does not necessanly adopt the 
by image 16, being displayed on the screen. y."'*"™ P~'°P°^ interconnecting network 
, . , ^ U-e., Internet 100) and is more geographically constrained 
An example of a browser program is the NaUonal Center j^an the larger network. The localized network may connect 
for Supercomputmg Application's (NCSA) Mosaic software lO i^gg, network through a "gateway" or "node" imple- 
developed by the University of Illinois at Urbana/ mented on, for example, a server. 
Champaign 111. Another example is "Cello" avaUable on the ^^^^^ j^^^^^^^j networks, such as 
Internet at http://wwwJaw.cornell.eduA Many viewere ex_^«t ^^^^ g jjO, which interconnects users 122, 124 and 126 
that handle various file formats such as TIF, ".GIF, ^^-^ respective computer systems to Internet 100. 
formats. When a browser program invokes a viewer 15 ^^^^^^^ ^ ^^^^ interconnected computer 
program, the viewer k launched as a separate process. The „^ communication links. Communication links 
view displays the full miage in a separate window" (in a ^ hardwire, fiber optic cable, satelUte or other radio 
windowmg environment) or on a separate screen This ^^^^ propagation, etc. Data may move from server A to 
meansthatthebrowser programisnolongeractive while the ^^^^ g (^^^^^ ^^^y^^ intermediate servers and 
viewer is active. By using md.catore to act as place holders 20 ^^^^^^-^^^^^ computers and data processing 
for fuU images that are retrieved and displayed only when a equipment not shown in FIG. 2 but symbolically represented 
user selects the indicator, data tramc over the network is ^ Internet 100 

reduced. Also, since the retrieval and display of large images . . i . i . j * 

, . f *u A user at a workstation or personal computer need not 

may require several seconds or more of transfer time the user *u t . * • i . l 

. \ . , -..u * c j.u* c connect to the Internet via a larger computer, such as server 

does not have to wait to have images transferred that are of 25 n t-u- • c lu n 

^ A or server B. This is shown, for example, by small 

no interest to the user. , ^-^ * j j- .i . t * finn u 

computer 130 connected directly to Internet 100 as by a 

Returning to FIG. 1, another type of data object is a sound telephone modem or other link. Also, a server need not have 

object shown as sound icon 24 within the hypermedia users connected to it locally, as is shown by server C at 132 

document. When the user selects sound icon 24, the user's piQ 2. Many configurations of large and small computers 

computer accesses sound data shown symbolically by data possible 

file 40. -nie accessed sound data plays through a speaker or Typically* a computer on the Internet is characterized as 

other audio device. ^^^^^^ ^ "client" or "server"* depending on the role that the 

As discussed above, hypermedia documents allow a user computer is playing with respect to requesting information 
to access different data objects. The objects may be text, ^5 or providing information. Client computers are computers 

images, sound files, video, additional documents, etc. As that typically request information from a server computer 

used in this specification, a data object is information which provides the information. For this reason, servers are 

capable of being retrieved and presented to a user of a usually larger and faster machines that have access to many 

computer system. Some data objects include executable data files, programs, etc., in a large storage associated with 

code combined with data. An example of such a combination the server. However, the role of a server may also be adopted 

is a "self-extracting" data object that includes code to by a smaller machine depending on the transaction. ITiat is, 

"unpack" or decompress data that has been compressed to user 110 may request information via their computer 108 

make it smaller before transferring. When a browser from server A. At a later time, server A may make a request 

retrieves an object such as a self -extracting data object the for information from computer 108. In the first case, where 

browser may allow the user to "launch" the self-extracting computer 108 issues a request for information from server A, 

data object to automatically execute the unpacking instruc- computer 108 is a "client" making a request of information 

tions to expand the data object to its original size. Such a from server A. Server A may have the information in a 

combination of executable code and data is limited in that storage device that is local to Server A or server A may have 

the user can do no more than invoke the code to perform a to make requests of other computer systems to obtain the 

singular function such as performing the self-extraction after information. User 110 may also request information via their 

which time the object is a standard data object. computer 108 firom a server, such as server B located at a 

Other existing approaches to embedding interactive pro- remote geographical location on the Internet. However, user 

gram objects in documents include the Object Linking and 110 may also request information from a computer, such as 

Embedding (OLE) facility in Microsoft Windows, by small computer 124, thus placing small computer 124 in the 
Microsoft Corp., and OpenDoc, by Apple Computer, Inc. At 55 role of a "server." For purposes of this specification, cUent 

least one shortcoming of these approaches is that neither is and server computers are categorized in terms of their 

capable of allowing a user to access embedded interactive predominant role as either an information requestor or 

program objects in distributed hypermedia documents over provider. Clients are generally information requestors, while 

networks. servers are generally information providers. 

no, 2 is an example of a computer network. In FIG. 2, 60 Referring again to FIG. 1, data objects such as distributed 

computer systems are connected to Internet 100, although in hypermedia documents 10, 12 and 14, image 16 and sound 

practice Internet 100 may be replaced by any suitable data file 40, may be located at any of the computers shown 

computer network. In FIG. 2, a user 102 operates a small in FIG. 2. Since these data objects may be linked to a 

computer 104, such as a personal computer or a work document located on another computer the Internet allows 
station. 'I'he user's computer is equipped with various 65 for remote object linking. 

components, such as user input devices (mouse, trackball. For example, hypertext document 10 of FIG. 1 may be 

keyboard, etc.), a display device (monitor, liquid crystal located at user 110*s client computer 108. When user 110 
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makes a request by, for example, clicking on hypertext 20 
(i.e., the phrase "hypermedia")) user llO's small client 
computer 108 processes links within hypertext document 10 
to retrieve document 14. In this example, we assume that 
document 14 is stored at a remote location on server B, 
Thus, in this example, computer 108 issues a command that 
includes the address of document 14. This command is 
routed through server A and Internet 100 and eventually is 
received by server B. Server B processes the command and 
locates document 14 on its local storage. Server 14 then 
transfers a copy of the document back to client 108 via 
Internet 100 and server A. After client computer 108 
receives document 14, it is displayed so that user 110 may 
view it. 

Similarly, image object 16 and sound data file 40 may 
reside at any of the computers shown in FIG. 2. Assuming 
image object 16 resides on server C when user 110 clicks on 
image icon 22, client computer 108 generates a command to 
retrieve image object 16 to server C. Server C receives the 
command and transfers a copy of image object 16 to client 
computer 108. Alternatively, an object, such as sound data 
file 40, may reside on server A so that it is not necessary to 
traverse long distances via the Internet in order to retrieve 
the data object. 

The Internet is said to provide an "open distributed 
hypermedia system." It is an "open" system since Internet 
100 implements a standard protocol that each of the con- 
necting computer systems, 106, 130, 120, 132 and 134 must 
implement (TCP/IP). It is a "hypermedia" system because it 
is able to handle hypermedia documents as described above 
via standards such as the HTTP and HTML hypertext 
transmission and mark up standards, respectively. Further, it 
is a "distributed" system because data objects that are 
imbedded within a document may be located on many of the 
computer systems connected to the Internet. An example of 
an open distributed hypermedia system is the so-called 
"world-wide web" implemented on the Internet and dis- 
cussed in papers such as the Bemers-Lee reference given 
above. 

The open distributed hypermedia system provided by the 
Internet allows users to easily access and retrieve different 
data objects located in remote geographic locations on the 
Internet. However, this open distributed hypermedia system 
as it currently exists has shortcomings in that today's large 
data objects are limited largely by bandwidth constraints in 
the various communication links in the Internet and local- 
ized networks, and by the limited processing power, or 
computing constraints, of small computer systems normally 
provided to most users. Large data objects are diflBcult to 
update at frame rates fast enough (e.g., 30 frames per 
second) to achieve smooth animation. Moreover, the pro- 
cessing power needed to perform the calculations to animate 
such images in real time does not exist on most 
workstations, not to mention personal computers. Today's 
browsers and viewers are not capable of perfonming the 
computation necessary to generate and render new views of 
these large data objects in real time. 

For example, the Internet's open distributed hypermedia 
system allows users to view still images. These images are 
simple non- interactive two-dimensional images, similar to 
photographs. Much digital data available today exists in the 
form of high -resolution multi-dimensional image data (e.g., 
three dimensional images) which is viewed on a computer 
while allowing the user to perform real time viewing trans- 
formations on the data in order for the user to better 
understand the data. 

An example of such type of data is in medical imaging 
where advanced scanning devices, such as Magnetic Reso- 
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nance Imaging (MRI) and Computed Tomography (CT), are 
widely used in the fields of medicine, quality assurance and 
meteorology to present physicians, technicians and meteo- 
rologists with large amounts of data in an efiEcient way. 

5 Because visualization of the data is the best way for a user 
to grasp the data's meaning, a variety of visualization 
techniques and real time computer graphics methods have 
been developed. However, these systems are bandwidth- 
intensive and compute-intensive and often require multipro- 

10 cessor arrays and other specialized graphics hardware to 
carry them out in real time. Also, large amounts of secondary 
storage for data are required. The expense of these require- 
ments has limited the ability of researchers to readily 
exchange findings since these larger computers required to 

15 store, present and manipulate images are not available to 
many of the researchers that need to have access to the data. 

On the other hand, small client computers in the form of 
personal computers or workstations such as client computer 
108 of FIG. 2 are generally available to a much larger 

20 number of researchers. Further, it is common for these 
smaller computers to be connected to the Internet. Thus, it 
is desirable to have a system that allows the accessing, 
display and manipulation of large amounts of data, espe- 
cially image data, over the Internet to a small, and relatively 

25 cheap, client computer. 

Due to the relatively low bandwidth of the Internet (as 
compared to today's large data objects) and the relatively 
small amount of processing power available at client 
computers, many valuable tasks performed by computers 
cannot be performed by users at client computers on the 
Internet. Also, while the present open distributed hyperme- 
dia system on the Internet allows users to locate and retrieve 
data objects it allows users very little, if any, interaction with 
these data objects. Users are limited to traditional hypertext 
and hypermedia forms of selecting linked data objects for 
retrieval and launching viewers or other forms of external 
software to have the data objects presented in a comprehen- 
sible way. 

Thus, it is desirable to have a system that allows a user at 
a small client computer connected to the Internet to locate, 
retrieve and manipulate data objects when the data objects 
are bandwidth-intensive and compute-intensive. Further, it 
is desirable to allow a user to manipulate data objects in an 
interactive way to provide the user with a better understand- 
ing of information presented and to allow the user to 
accomplish a wider variety of tasks. 

SUMMARY OF THE INVENTION 

50 The present invention provides a method for running 
embedded program objects in a computer network environ- 
ment. The method includes the steps of providing at least 
one client workstation and one network server coupled to the 
network environment where the network environment is a 

55 distributed hypermedia environment; displaying, on the cli- 
ent workstation, a portion of a hypermedia document 
received over the network from the server, where the hyper- 
media document includes an embedded controllable appli- 
cation; and interactively controlling the embedded control- 

60 lable application from the client workstation via 
communication sent over the distributed hypermedia envi- 
ronment. 

The present invention allows a user at a client computer 
connected to a network to locate, retrieve and manipulate 
65 objects in an interactive way. The invention not only allows 
the user to use a hypermedia format to locate and retrieve 
program objects, but also allows the user to interact with an 
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application program located at a remote computer. Interpro- FIG. 10 is a diagram of the various processes and data 

cess communication between the hypermedia browser and paths in the present invention, 

the embedded application program is ongoing after the DETAIT ED DESCRIPTION OF A PREFERRED 

program object has been launched. The user is able to use a " EMBODIMENT 

vast amount of computing power beyond that which is 5 ^ ^ . /i l * * 

^r.^t.ir.^A th^ ^i;<.„t\.^«,«..t^; 375 pages of Source code on 4 microfiche Appendices A 

contained m the user s client computer. i o • i ^ * *u* r *• n-u i 

and B are provided to this specification. Ine source code 

In one apphcation, high resolution three dimensional should be consulted to provide details of a specific embodi- 

images are processed in a distributed manner by several meat of the invention in conjunction with the discussion of 

computers located remotely from the user's client computer. the routines in this specification. The source code in Appen- 
This amounts to providing parallel distributed processing for ^ 0 ^ includes NCSA Mosaic version 2.4 source code along 

tasks such as volume rendering or three dimensional image with modifications to the source code to implement the 

transformation and display. Also, the user is able to rotate, present invention. Appendix B includes source code imple- 

scale and otherwise reposition the viewpoint with respect to menling an application program interface. ^Fhe source code 

these images without exiting the hypermedia browser soft- jg written in the "C" computer language to run on an 
ware. The control and interaction of viewing the image may is X- Window platform. 

be provided within the same window that the browser is pjc. 3 is an illustration of a computer system suitable for 
using assuming the environment is a "windowing" environ- ^-^^^ jhe present invention, FIG, 3 depicts but one 
ment. The viewing transformation and volume rendering example of many possible computer types or configurations 
calculations may be performed by remote distributed com- ^^p^^le of being used with the present invention. FIG. 3 
puter systems. shows computer system 150 including display device 153, 
Once an image representing a new viewpoint is computed display screen 155, cabinet 157, keyboard 159 and mouse 
the frame image is transmitted over the network to the user's 161. Mouse 161 and keyboard 159 are "user input devices." 
client computer where it is displayed at a designated position Other examples of user input devices are a touch screen, 
within a hypermedia document. By transmitting only light pen, track ball, data glove, etc. 
enough information to update the image, the need for a high Mouse 161 may have one or more buttons such as buttons 
bandwidth data connection is reduced. Compression can be 163 shown in FIG, 3. Cabinet 157 houses familiar computer 
used to further reduce the bandwidth requirements for data components such as disk drives, a processor, storage means, 
transmission. etc. As used in this specification "storage means" includes 
Other applications of the invention are possible. For any storage device used in connection with a computer 
example, the user can operate a spreadsheet program that is system such as disk drives, magnetic tape, solid state 
being executed by one or more other computer systems memory, bubble memory, etc. Cabinet 157 may include 
connected via the network to the user's client computer. additional hardware such as input/output (I/O) interface 
Once the spreadsheet program has calculated results, the cards for connecting computer system 150 to external 
results may be sent over the network to the user's client devices such as an optical character reader, external storage 
computer for display to the user. In this way, computer devices, other computers or additional devices, 
systems located remotely on the network can be used to FIG, 4 is an illustration of basic subsystems in computer 
provide the computing power that may be required for system 150 of FIG. 3. In FIG. 4, subsystems are represented 
certain tasks and to reduce the data bandwidth by only by blocks such as central processor 180, system memory 181 
transmitting results of the computations. consisting of random access memory (RAM) and/or read- 
Still other applications of the present invention are only memory (ROM), display adapter 182, monitor 183 
possible, as disclosed in the specification, below. (equivalent to display device 153 of FIG. 3), etc. The 

subsystems are interconnected via a system bus 184. Addi- 

BRIEF DESCRIPTION OF TI IE DRAWINGS tional subsystems such as a printer, keyboard, fixed disk and 

™^ 11,*. 1 fu * . Au A' others are shown. Peripherals and input/output (I/O) devices 

FIG. 1 illustrates examples of hypertext and hypermedia . . j * X * * u \ i 

- t H r V can be connected to the computer system by, for example 

documents and imks; ^^.^^ p^^^ example, serial port 185 can be used to 

FIG. 2 IS an example of a computer network; connect the computer system to a modem for connection to 

FIG. 3 is an illustration of a computer system suitable for a network or serial port 185 can be used to interface with a 

use with the present invention; mouse input device. The interconnection via system bus 184 

FIG. 4 is an illustration of basic subsystems in the allows central processor 180 to communicate with each 

computer system of FIG. 3; subsystem and to control the execution of instructions from 

no. 5 is an illustration of an embodiment of the invention system memory 181 or fixed disk 186, and the exchange of 

using a client computer, server computer and a network; information between subsystems. Other arrangements of 

, , . u J- * r*u subsystems and interconnections are possible. 

FIG. 6 shows another embodiment of the present mven- 55 ^ . r . , . a. - 

tion using additional computers on the network; P'^- » J* «° tUustrationof an embodiment of the mvention 

„^ . . . „ , ^. ^ . ^ . ... using a client computer, server computer and a network. 

K uwt ' fil functionahty within 5^ ^^^^^ ^^^^^^^^ ^00 communicates with server 

theHlMLparse.cme, computer 204 via network 206. Both client computer 200 

nG,7Bisaflowchartofsomeof the functionahty within server computer 204 use a network protocol layer to 

the HTMLformat,c file; communicate with network 206. In a preferred embodiment. 

FIG, 8A is a flowchart of some of the functionality within network 206 is the Internet and the network protocol layers 

the HTMLwidget.c file; are TCP/IP. Other networks and network protocols may be 

FIG, 8B is a flowchart of some of the functionality within used. For ease of illustration, additional hardware and soft- 
the HTML.c file; 65 ware layers are not shown in FIG, 5, 

FIG. 9 is a screen display generated in accordance with Client computer 200 includes processes, such as browser 

the present invention; and client 208 and application client 210, In a preferred 
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embodiment, application client 210 is resident within client to display the multidimensional embryo data on the display 

computer 200 prior to browser client 208's parsing of a screen to a user of the client computer 200. The user is then 

hypermedia document as discussed below. In a preferred able to interactively operate controls to recompute different 

embodiment application client 210 resides on the hard disk views for the image daU. In a preferred embodiment, a 
or RAM of client computer 200 and is loaded (if necessary) 5 control window is displayed within, or adjacent to, a win- 

and executed when browser client 208 detects a link to dow generated by browser client 208 that conuins a display 

application client 210. The preferred embodiment uses the °^ hypermedia document 212. An example of such display 

XEvent interprocess communication protocol to exchange discussed below in connection with FIG. 9. Thus, the user 

information between browser client 208 and application is able to mteractively mampulate a multidimensional unage 
cUent 210 as described in more detail, below. Another 30 object by means of the present mvention. In order to make 

possibility is to install application client 210 as a "terminate plication client 2io integral with displays created by 

and stay resident" (TSR) program in an operating system browser chent 208 both the browser cbent and the apph"- 

environment, such as X-Window. Thereby making access to <=f °° f ^ " communicalion with each other, as 

application client 210 much faster. the arrow connecting the two withm client com- 
„ . , o ic puter 200. The manner of communication is through an 

Browser chent 208 is a process that a user of client is ^^^^j^^ ^ ^^^^^^ ^^^^ discussed below 

computer 200 invokes m order to access various data ^ , . ^ • u kt/-^o a * 

, . , u u J- J * ^ ! ift^- Browser client 208 is a process, such as NCSA Mosaic, 

obiects, such as hypermedia documents, on network 206. ^ ^ * i- i- . -^i/i • u ^ • tu. 

J- . -^^/^ L 1- . . Cello, etc. Application chent 210 IS embodied m software 

Hypermedia document 212 shown within Client computer \, j j i . n ^ «i/io.. ^ «n i" 

• 1 r L J- J * L- * * presently under development called VIS and Panel 

200 IS an example of a hypermedia document, or object, that . j u *i. . r ^ j km . * 

. .J * f .u- \ u \i' m created by the Center for Knowledge Management at the 

a user has requested access to. In this example, hypermedia 20 ^^.^^^.^^ of California. San Francisco, as part of the Doyle 

document 212 has been retrieved from a server connected to ^ distributed hypermedia object embedding approach 

loaded mto. e.g., chent computer J^^^ "Integraled Control of Distributed Volume 

200 s RAM or other storage device. VisuaUzation TTirough the World-Wide-Web," by C. Ang. D. 

Once hypermedia document 212 has been loaded mto Martin, M. Doyle; to be published in the Proceedings of 

cUent computer 200, browser client 208 parses hypermedia VisuaUzation 1994, IEEE Press, Washington, D.C., October 

document 212. In parsing hypermedia document 212, ^994 

browser client 208 delects Unks to data objects as discussed y^^j^^^ descriptions of software embodying the 

above in the Background of the Invention section. In FIG. 5, ^^^^^j j^^^^.j^^ ^^^^^ ^^^ji^^^le as hyperlinked data 

hypennedia document 212 includes an embedded program ^^^-^^ ^^^^ y^^^^ g^^ryo Project's World Wide Web 

hnkat 214. Emb edded propram link ?14 iHp.nlifips iipphra . document at the URL address "HTTP://visembryo. 

tion client 212 as an application to invoke . In this present edu/" 

example, the application, namely, application client 210 ^^bodiment of the present invention uses an 

resides on the same computer as the browser client 208 that ^. J ia.i 

. . • u J- J . application server process executing on server computer 204 

the user is executing to view the hypermedia document. . ■ * • ■ *u * ^ * u ^ au 
„ . J V , -^-f.* • 1 J jj*- 1 35 to assist m processing that may need to be performed by an 

Embedded program link 214 may include additional ^ . *^ i - t-t>- e r *• 

. ^ ^ ^ „ 1- * external program. For example, in FIG. 5, application server 

information, such as parameters, that tell application client Jlc ^nA a t;/ot;^« e«r,.«r 'y'jn 

--rt, * , ! . .j^^ ,. , 220 resides on server computer 204. Application server 220 

210 how to proceed. For example, embedded program hnk ^^^^ ^ communication with appUcation client 210 residing 

214 may mclude a specification as to a data object that ^y^^, jOO. In a preferred embodiment, appli- 

application client 210 is to retrieve and process, . i-^ • n a \rnc i rL.,i^ 

^ 40 cation server 220 is called VRServer, also a part of Doyle 

When browser client 208 encounters embedded program q^^^^,^ approach. Since server computer 204 is typically a 

Unk 214, it invokes application client 210 (opUonally, with i^^g^j computer having more data processing capabilities 

parameters or other information) and appUcation client 210 ^^^^^^ ^^^^^^^ capacity, application server 220 can 

executes instructions to perform processing in accordance ^^^^^^^ ^^^^ eflBcienUy, and much faster, than application 

with the present invention. ^^^^^^ 210 in executing complicated and numerous instruc- 

An example of the type of processing that application tions. 

client 210 may perform is multidimensional image visual- present example where a multidimensional image 

ization. Note that application client 210 is in communication ^^^^^^ representing medical data for an embryo is being 

with network 206 via the network protocol layer of client viewed, application server 220 could perform much of the 

computer 200. This means that application client 210 can viewing transformation and volume rendering calculations 

make requests over network 206 for data objects, such as ^Uow a user to interactively view the embryo data at their 

multidimensional image objects. For example, application client computer display screen. In a preferred embodiment, 

client 210 may request an object, such as object 1 at 216, application client 210 receives signals from a user input 

located in server computer 204. Application client 210 may j^e at the user's client computer 200. An example of such 
make the request by any suitable means. Assuming network 55 i^put would be to rotate the embryo image from a current 

206 is the Internet, such a request would typicaUy be made position to a new position from the user's point of view. This 

by using HTTP in response to a HTML-style link definition information is received by application client 210 and pro- 

for embedded program link 214. ^.^^^ to generate a command sent over network 206 to 

Assuming application client 210 has made a request for application server 220. Once application server 220 receives 
the data object at 216, server process 218 ultimately receives eo the information in the form of, e.g., a coordinate transfor- 

the request. Server process 218 then retrieves data object mation for a new viewing position, application server 220 

216 and transfers it over network 206 back to application performs the mathematical calculations to compute a new 

client 210, To continue with the example of a multidimen- view for the embryo image. Once the new view has been 

sional visualization application, data object 216 may be a computed, the image data for the new view is sent over 
threedimensionalviewofmedicaldatafor, e.g., an embryo. ^5 network 206 to application client 210 so that application 

After application client 210 receives the multidimensional client 210 can update the viewing window currently dis- 

data object 216, application client 210 executes instructions playing the embryo image. In a preferred embodiment, 
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application server 220 computes a frame buffer of raster 
display data, e.g., pixel values, and transfers this frame 
buffer to application client 210. Techniques, such as data 
compression and delta encoding, can be used to compress 
the data before transmitting over network 206 to reduce the 
bandwidth requirement. 

It will be readily seen that application server 220 can 
advantageously use server computer 204's computing 
resources to perform the viewing transformation much more 
quickly than could application client 210 executing on client 
computer 200. Further, by only transmitting the updated 
frame buffer containing a new view for the embryo image, 
the amount of data sent over network 206 is reduced. By 
using appropriate compression techniques, such as, e.g., 
MPEG (Motion Picture Experts Group) or JPEG (Joint 
Photographic Experts Group), efficient use of network 206 is 
preserved. 

FIG. 6 shows yet another embodiment of the present 
invention. FIG. 6 is similar to FIG. 5, except that additional 
computers 222 and 224 are illustrated. Each additional 
computer includes a process labeled "Application 
(Distributed)." The distributed application performs a por- 
tion of the task that an application, such as application server 
220 or application client 210, perform. In the present, 
example, tasks such as volume rendering may be broken up 
and easily performed among two or more computers. These 
computers can be remote from each other on network 206. 
Thus, several computers, such as server computer 204 and 
additional computers 222 and 224 can all work together to 
perform the task of computing a new viewpoint and frame 
buffer for the embryo for the new orientation of the embryo 
image in the present example. The coordination of the 
distributed processing can be performed at client computer 
200 by application client 210, at server computer 204 by 
application server 220, or by any of the distributed appli- 
cations executing on additional computers, such as 222 and 
224. In a preferred embodiment, distributed processing is 
coordinated by a program called "VIS" represented by 
application client 210 in FIG. 6. 

Other applications of the invention are possible. For 
example, the user can operate a spreadsheet program that is 
being executed by one or more other computer systems 
connected via the network to the user's client computer. 
Once the spreadsheet program has calculated results, those 
results may be sent over the network to the user's client 
computer for display within the hypermedia document on 
the user's client computer. In this way, computer systems 
located remotely on the network can be used to provide the 
computing power that may be required for certain tasks and 
to reduce the data bandwidth required by only transmitting 
results of the computations. 

Another type of possible application of this invention 
would involve embedding a program which runs only on the 
client machine, but which provides the user with more 
functionality than exists in the hypermedia browser alone. 
An example of this is an embedded client application which 
is capable of viewing and interacting with images which 
have been processed with Dr. Doyle's MetaMAP invention 
(U.S. Pat. No. 4,847,604). This MetaMAP process uses 
object-oriented color map processing to allow individual 
color index ranges within paletted images to have object 
identities, and is useful for the creation of, for example, 
interactive picture atlases. It is a more efficient means for 
defining irregular "hotspots" on images than the ISMAP 
function of the World Wide Web, which uses polygonal 
outlines to define objects in images. A MetaMAP -capable 
client-based image browser application can be embedded. 
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together with an associated image, within a hypermedia 
document, allowing objects within the MetaMAP-processed 
image to have URL addresses associated with them. When 
a user clicks with a mouse upon an object within the 

5 MetaMAP-processed image, the MetaMAP client applica- 
tion relays the relevant URL back to the hypermedia browser 
application, which then retrieves the HTML file or hyper- 
media object which corresponds to that URL. 
The various processes in the system of the present inven- 

10 tion communicate through a custom API called Mosaic/ 
External Application Program Interface MEAPI. The 
MEAPI set of predefined messages includes those shown in 
Table I. 

15 TABLE I 

Message Function Message Name 

Messages from server to client: 

20 1. Server Update Done XtNrcfresh Notify 

2. Server Ready XtNpanelStartNotify 

3. Server Exiting XtNpanelExitNotify 

Messages from client to server: 

4. Area Shown XtNmapNotify 
25 5. Area Hidden XtNunmapNotify 

6. Area Destroyed XtNexit Notify 



The messages in Table I are defined in the file protocol_lib,h 
in Appendix B, The functions of the MEAPI are provided in 

30 protocol_lib.c of Appendix B. Thus, by using MEAPI a 
server process communicates to a client application program 
to let the client application know when the server has 
finished updating information, such as an image frame 
buffer, or pixmap (Message 1); when the server is ready to 

35 start processing messages (Message 2) and when the server 
is exiting or stopping computation related to the server 
application program. 

For client to server communications, MEAPI provides for 
the client informing the server when the image display 

40 window area is visible, when the area is hidden and when the 
area is destroyed. Such information allows the server to 
decide whether to allocate computing resources for, e.g., 
rendering and viewing transformation tasks where the server 
is running an application program to generate new views of 

45 a multi dimensional object. Source code for MEAPI funda- 
mental functions such as handle_client_msg, register_ 
client, register_client_msg_callback and send_client„ 
msg may be found in protocoUib.c as part of the source 
code in Appendix B. 

50 Next, a discussion of the software processes that perform 
parsing of a hypermedia document and launching of an 
application program is provided in connection with Table II 
and FIGS. 7A, 7B, 8A and «B. 
Table II, below, shows an example of an HTML tag 

55 format used by the present invention to embed a link to an 
application program within a hypermedia document. 



TABLE II 



60 



<EMBED 




TYPE » ' 


'type" 


HREF - 


"href 


WIDTH - width 


HEIGHT 


" height 


> 





65 

As shown in Table II, the EMBED tag includes ^I'YPE, 
HREF, WIDTH and HEIGHT elements. The TYPE element 
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is a Multipurpose Internet Mail Extensions (MIME) type. 
Examples of values for the TYPE element are "application/ 
x-vis" or "video/mpeg". The type "application /x-vis" indi- 
cates that an application named "x-vis" is to be used to 
handle the object at the URL specified by the HREF. Other 5 
types are possible such as "application/x-inventor", 
"application/postscript" etc. In the case where TYPE is 
"apphcation/x-vis" this means that the object at the URL 
address is a three dimensional image object since the pro- 
gram "x-vis" is a data visualization tool designed to operate lO 
on three dimensional image objects. However, any manner 
of application program may be specified by the TYPE 
element so that other types of applications, such as a 
spreadsheet program, database program, word processor, 
etc. may be used with the present invention. Accordingly, the is 
object reference by the HREF element would be, 
respectively, ;a spreadsheet object, database object, word 
processor document object, etc. 

On the other hand, TYPE values such as "video/mpeg", 
"image/gif , "video/x-sgi-movie", etc. describe the type of 20 
data that HREF specifies. This is useful where an external 
application program, such as a video player, needs to know 
what format the data is in, or where the browser client needs 
to determine which application to launch based on the data 
format, llius, the ITPE value can specify either an apph- 25 
cation program or a data type. Other TYPE values are 
possible. HREF specifies a URL address as discussed above 
for a data object. Where TYPE is "application/x-vis" the 
URL address specifies a multi-dimensional image object. 
Where TYPE is "video/mpeg" the URL address specifies a 30 
video object. 

WIDTH and HEIGHT elements specify the width and 
height dimensions, respectively, of a Distributed Hyperme- 
dia Object Embedding (DHOE) window to display an exter- 
nal application object such as the three dimensional image 35 
object or video object discussed above. 

FIG. 7 A is a flowchart describing some of the function- 
ality within the HTMLparse.c file of routines. The routines 
in HTMLparse.c perform the task of parsing a hypermedia 
document and detecting the EMBED tag. In a preferred 40 
embodiment, the enhancements to include the EMBED tag 
are made to an HTML library included in public domain 
NCSA Mosaic version 2.4. Note that much of the source 
code in is pre-existing NCSA Mosaic code. Only those 
portions of the source code that relate to the new function- 45 
ality discussed in this specification should be considered as 
part of the invention. The new functionality is identifiable as 
being set off from the main body of source code by condi- 
tional compilation macros such as "#ifdef . . . #endif * as will 
be readily apparent to one of skill in the art. so 

In general, the flowcharts in this specification illustrate 
one or more software routines executing in a computer 
system such as computer system 1 of FIG, 1. The routines 
may be implemented by any means as is known in the art. 
For example, any number of computer programming ss 
languages, such as "C", Pascal, FORTRAN, assembly 
language, etc., may be used. Further, various programming 
approaches such as procedural, object oriented or artificial 
intelligence techniques may be employed. 

The steps of the flowcharts may be implemented by one 60 
or more software routines, processes, subroutines, modules, 
etc. It will be apparent that each flowchart is illustrative of 
merely the broad logical flow of the method of the present 
invention and that steps may be added to, or taken away 
from, the flowcharts without departing from the scope of the 65 
invention. Further, the order of execution of steps in the 
flowcharts may be changed without departing from the 
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scope of the invention. Additional considerations in imple- 
menting the method described by the flowchart in software 
may dictate changes in the selection and order of steps. 
Some considerations are event handling by interrupt driven, 
polled, or other schemes. A multiprocessing or multitasking 
environment could allow steps to be executed "concur- 
rently." For ease of discussion the implementation of each 
flowchart may be referred to as if implemented in a single 
"routine". 

The modifications to NCSA Mosaic version 2.4 software 
files HTMLparse.c, HTMLformat.c, HTMLwidget.c and 
HTML.c will next be discussed, in turn. 

Returning to FIG, 7, it is assumed that a hypermedia 
document has been obtained at a user's client computer and 
that a browser program executing on the client computer 
displays the document and calls a first routine in the HTM- 
Lparse.c file called "HTMLparse". This first routine, 
HTMLparse, is entered at step 252 where a pointer to the 
start of the document portion is passed. Steps 254, 256 and 
258 represent a loop where the document is parsed or 
scanned for HTML tags or other symbols. While the file 
HTMLparse.c includes routines to handle all possible lags 
and symbols that may be encountered, FIG. 7 A, for 
simplicity, only illustrates the handling of EMBED tags. 

Assuming there is more text to parse, execution proceeds 
to step 256 where routines in HTMLparse.c obtain the next 
item (e.g., word, tag or symbol) from the document. At step 
258 a check is made as to whether the current tag is the 
EMBED tag. If not, execution returns to step 254 where the 
next tag in the document is obtained. If, at step 258, it is 
determined that the tag is the EMBED tag, execution pro- 
ceeds to step 260 where an enumerated type is assigned for 
the tag. Each occurrence of a valid EMBED tag specifies an 
embedded object. HTMLParse calls a routine "get_mark" 
in HTMLparse.c to put sections of HTML document text 
into a "markup" text data structure. Routine get__mark, in 
turn, calls ParseMarkType to assign an enumerated type. The 
enumerated type is an identifier with a unique integer 
associated with it that is used in later processing described 
below. 

Once all of the hypermedia text in the text portion to be 
displayed has been parsed, execution of HTMLparse.c rou- 
tines terminates at step 262. 

FIG. 7B is a flowchart of routines in file HTMLformat.c 
to process the enumerated type created for the EMBED tag 
by routines in HTMLparse.c. In the X- Window implemen- 
tation of a preferred embodiment, the enumerated type is 
processed as if it is a regular Motif/XT widget. For details 
on X-Window development see, e.g., "Xlib Programming 
Manual," "X Toolkit Intrinsics Programming Manual" and 
"Motif Programming Manual" published by O'Reilly & 
Associates, Inc. HTMLformat is entered at step 270 where 
a pointer to the enumerated type to process is passed. 

At step 272 the parameters of the structure are initialized 
in preparation for inserting a DrawingArea widget on an 
HT^L page. This includes providing values for the width 
and height of a window on the display to contain an image, 
position of the window, style, URL of the image object, etc. 
Various codes are also added by routines in HTMLformat.c 
(such as TriggerMarkChanges) to insert an internal repre- 
sentation of the HTML statement into an object list main- 
tained internally by the browser. In the X-Window applica- 
tion corresponding to the source code of Appendix A, the 
browser is NCSA Mosaic version 2.4. 

FIG. 8A is a flowchart for routine HTMLwidget. HTM- 
Lwidget creates display data structures and launches an 
external application program to handle the data object 
specified by the URL in the EMBED tag. 
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HTMLwidget is entered at step 280 after HTMLformat data and transfers the pixmap data (frame image data) to a 

has created the internal object representation of the EMBED buffer to which the browser has access. The browser only 

tag. HTMLwidget is passed the internal object and performs needs to respond to the refresh request to copy the contents 

its processing on the object. At step 282 the DrawingArea from the updated pixmap to the DrawingAiea. The Panel 

widget is created according to the type of the internal 5 process sends messages as "Msg" sending performed by 

representation, from HTMLformat, specified in the internal routines such as vis„send_msg and vis_handle panel_msg 

object. Similarly, at step 284 a pixmap area for backing send events (mousemove, keypress, etc.) to the external 

storage is defined. appUcation. 

At step 286 a check is made as to whether the type . ^ ^ f^^^^ ^^^P.^^^ ^.^ the mvention showing an 

attribute of the object, i.e., the value for the ITPE element lO ^^^^^f f^^^ application object (m this case a three dimen- 

th^ civynacn :o . if ioa o sional image object) in a wmdow within a browser window. 

of the EMBED tag, is an application, f so, step 290 is ,^ p,^ ^^^^^ ^^^^^^^ ^^SA Mosaic version 2.4. The 

executed to aunch a predetermined application. In a pre- ^^^^^ p^^^j ^j^^^^^^ ^^^^ ^.^^^^^ 

ferred embodiment an application ls aunched accordingto ^^ove. HG. 9 shows screen display 356 Mosaic window 350 

a user-dehned list of application type/application pairs. The containing image window 352 and a portion of a panel 

list IS defined as a user-configurable XResource as described is ^^indow 354. Note that image window 352 is within Mosaic 

in "Xhb Programming Manual." An alternative embodiment window 350 while panel window 354 is external to Mosaic 

could use the MIME database as the source of the list of window 350. Another possibility is to have panel window 

application type/application pairs. The routine "vis_start_ 354 within Mosaic window 350. By using the controls in 

external_application" in file HTMLformat. c is invoked to panel window 354 the user is able to manipulate the image 

match the application type and to identify the application to 20 within image window 352 in real time do perform such 

launch. operations as scaling, rotation, translation, color map 

The external application is started as a child process of the selection, etc. In FIG. 9, two Mosaic windows are being used 

current running process (Mosaic), and informed about the to show two different views of an embryo image. One of the 

window ID of the DrawingArea created in HTMLformat. views is rotated by six degrees from the other view so that 

The external application is also passed information about the 25 ^ stereoscopic effect can be achieved when viewing the 

ID of the pixmap, the data URL and dimensions. Codes for images. Communication between Panel and VIS is via 

communication such as popping-up/iconifying, start "Tooltalk" described in, e.g., "Tooltalk LLl Reference 

notification, quit notification and refresh notification with ^}!°^°^'-. , , ^ . . 

external applications and DrawingArea refreshing are also , ™- 'l^"" illustration of the processes VIS Panel and 

added. Examples of such codesVre (1) "setup/start" in 30 Y^^'"^'' ^^^^^^ ^^'-^^^^ ^ ^^^^l^ ^l^iu h ^ ^ 

, . . , 1 • J • jjrj^A browser process, Mosaic, communicates with the Panel 

yis register client and vis_^et panel_window in HTM- ^^^^^ inter-client communication mechanisms such as 

Lwidgels.c; (2) handle messages from external apphca- ^^^^^^^ t^e X-Window environment. The Panel process 

tions m vis__handle panel„msg m HTMLwidgets.c; (3) communicates with the VIS process through a communica- 

"send messages to external applications" m vis„send_msg tions protocol (ToolTalk, in the preferred embodiment) to 

m HTMLwidgets.c; (4) "terminate external applications" in 35 exchange visualization command messages and image data. 

vis_exit in HTMLwidgets.c which calls vis_send_msg to The image data is computed by one or more copies of a 

send a quit message; and (5) "respond to refresh msgs" in process called VRServer that may be executing on remote 

vis_redraw in HTMLwidgets.c. computers on the network. VRServer processes respond to 

If, at step 286, the type is determined not to be an requests such as rendering requests to generate image seg- 

application object (e.g., a three dimensional image object in 40 ments. The image segments are sent to VIS and combined 

the case of application "x-vis") a check is made at step 288 into a pixmap, or frame image, by VIS. The frame image is 

to determine if the type is a video object. If so, step 292 is then transferred to the Mosaic screen via communications 

executed to launch a video player application. Parameters between VIS, Panel and Mosaic. A further description of the 

are passed to the video player application to allow the player ^^1^ transfer may be found in the paper "Integrated Control 

to display the video object within the DrawingArea within 45 ^f Distributed Volume Visualization ITirough the World- 

the display of the portion of hypermedia document on the Wide-Web " referenced above. 

cUenfs computer. Note that many other application objects , foregomg specification, the invention has been 

types are possible as described above. ^''^''I'^^f f^'Tl '° ^ '^'u exemplary embodi- 

HG. 8B is a flowchart for routine HTML. Routine HTML ^*^°1 l^^of- ^ will, however be evident that various 

,1 r« u J » *!. u- * J * . modifications and changes may be made thereunto without 

akes care of shuttmg down the objects, data areas, etc 50 ^ . ^^^^ ^^J^^^ .^^ .^^^^^-^^ 

that were set up to launch the external apphcation and ^rth in the appended claims. For example, various 

display the data object. HI ML is entered at step 300 and is programming languages and techniques can be used to 

called when the display or other processing of the EMBED implement the disclosed invention. Also, the specific logic 

tag has been completed. At step 302 the display window is presented to accomplish tasks within the present invention 

removed and the memory areas for the pixmap and internal ss may be modified without departing from the scope of the 

object structure is made free for other uses. Completion of invention. Many such changes or modifications will be 

processing can be by user command or by computer control. readily apparent to one of ordinary skill in the art. The 

The present invention allows a user to have interactive specification and drawings are, accordingly, to be regarded 

control over application objects such as three dimensional in an illustrative rather than a restrictive sense, the invention 

image objects and video objects. In a preferred embodiment, 60 being limited only by the provided claims, 

controls are provided on the external applications* user What is claimed is: 

interface. In the case of a VIS/panel application, a process, 1. A method for running an application program in a 

"panel" creates a graphical user interface (GUI) thru which computer network environment, comprising: 

the user interacts with the data. The application program, providing at least one client workstation and one network 

VIS, can be executing locally with the user's computer or 65 server coupled to said network environment, wherein 

remotely on a server, or on one or more diflerent computers, said network environment is a distributed hypermedia 

on the network. The application program updates pixmap environment; 
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executing, at said client workstation, a browser 
application, that parses a first distributed hypermedia 
document to identify text formats included in said 
distributed hypermedia document and for responding to 
predetermined text formats to initiate processing speci- 
fied by said text formats; utilizing said browser to 
display, on said client workstation, at least a portion of 
a first hypermedia document received over said net- 
work from said server, wherein the portion of said first 
hypermedia document is displayed within a first 
browser-controlled window on said client workstation, 
wherein said first distributed hypermedia document 
includes an embed text format, located at a first location 
in said first distributed hypermedia document, that 
specifies the location of at least a portion of an object 
external to the first distributed hypermedia document, 
wherein said object has type information associated 
with it utilized by said browser to identify and locate an 
executable application external to the first distributed 
hypermedia document, and wherein said embed text 
format is parsed by said browser to automatically 
invoke said executable application to execute on said 
client workstation in order to display said object and 
enable interactive processing of said object within a 
display area created at said first location within the 
portion of said first distributed hypermedia document 
being displayed in said first browser-controlled win- 
dow. 

2. The method of claim 1, wherein said executable apph- 
cation is a controllable application and further comprising 
the step of; 

interactively controlling said controllable application on 
said client workstation via inter-process communica- 
tions between said browser and said controllable appli- 
cation. 

3. The method of claim 2, wherein the communications to 
interactively control said controllable application continue 
to be exchanged between the controllable application and 
the browser even after the controllable application program 
has been launched. 

4. The method of claim 3, wherein additional instructions 
for controlling said controllable application reside on said 
network server, wherein said step of interactively controlling 
said controllable application includes the following sub- 
steps: 

issuing, from the client workstation, one or more com- 
mands to the network server; 

executing, on the network server, one or more instmctions 
in response to said commands; 

sending information from said network server to said 
client workstation in response to said executed instruc- 
tions; and processing said information at the client 
workstation to interactively control said controllable 
application. 

5. The method of claim 4, wherein said additional instruc- 
tions for controlling said controllable application reside on 
said client workstation. 

6. A computer program product for use in a system having 
at least one client workstation and one network server 
coupled to said network environment, wherein said network 
environment is a distributed hypermedia environment, the 
computer program product comprising: 

a computer usable medium having computer readable 
program code physically embodied therein, said com- 
puter program product further comprising: 
computer readable program code for causing said client 
workstation to execute a browser application to parse 
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a first distributed hypermedia document to identify 
text formats included in said distributed hypermedia 
document and to respond to predetermined text for- 
mats to initiate processes specified by said text 

5 formats; 

computer readable program code for causing said client 
workstation to utilize said browser to display, on said 
client workstation, at least a portion of a first hyper- 
media document received over said network from 
said server, wherein the portion of said first hyper- 
media document is displayed within a first browser- 
controlled window on said client workstation, 
wherein said first distributed hypermedia document 

J 5 includes an embed text format, located at a first 

location in said first distributed hypermedia 
document, that specifies the location of at least a 
portion of an object external to the first distributed 
hypermedia document, wherein said object has type 

20 information associated with it utilized by said 

browser to identify and locate an executable appli- 
cation external to the first distributed hypermedia 
document, and wherein said embed text format is 
parsed by said browser to automatically invoke said 

25 executable application to execute on said client 

workstation in order to display said object and enable 
interactive processing of said object within a display 
area created at said first location within the portion of 
said first distributed hypermedia document being 

30 displayed in said first browser-controlled window. 

7. The computer program product of claim 6, wherein said 
executable application is a controllable apphcation and 
further comprising: 

computer readable program code for causing said client 
35 workstation to interactively control said controllable 
apphcation on said client workstation via inter-process 
communications between said browser and said con- 
trollable application. 

8. The computer program product of claim 7, wherein the 
40 communications to interactively control said controllable 

application continue to be exchanged between the control- 
lable application and the browser even after the controllable 
application program has been launched. 

9. The computer program product of claim 8, wherein 
45 additional instructions for controlling said controllable 

application reside on said network server, wherein said step 
of interactively controlling said controllable application 
includes: 

computer readable program code for causing said client 
workstation to issue, from the client workstation, one or 
more commands to the network server; 

computer readable program code for causing said network 
server to execute one or more instructions in response 
to said commands; 

computer readable program code for causing said network 
sever to send information to said cUcnt workstation in 
response to said executed instructions; and 

computer readable program code for causing said client 
60 workstation to process said information at the client 
workstation to interactively control said controllable 
application. 

10. The computer program product of claim 9, wherein 
said additional instructions for controlling said controllable 

65 application reside on said client workstation. 
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